Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize Qwen2VL vision model by precomputing cos/sin embeds before ViT blocks #35837

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Optimize Qwen2VL vision model by precomputing cos/sin embeds before ViT blocks #35837

wants to merge 3 commits into from
+211 −72

Conversation

li-plus
Copy link
Contributor

@li-plus li-plus commented Jan 22, 2025

What does this PR do?

The current implementation of Qwen2VL recomputes cos/sin embeddings and repeats them on each layer of the ViT model. This is computational inefficient. This PR attempts to precompute the rotary embeddings at the beginning of the ViT forward function to save computation and memory.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts @qubvel

@qubvel
Copy link
Member

qubvel commented Jan 22, 2025

cc @zucchini-nlp for vlms

zucchini-nlp
zucchini-nlp reviewed Jan 23, 2025
View reviewed changes
Copy link
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes total sense, thanks! One tiny nit, we'd need to keep backwards compatibility for the vision attention classes and add a small deprecation warning

As an example of how it was in llama:

transformers/src/transformers/models/llama/modeling_llama.py

Lines 347 to 356 in c2820c9

if position_embeddings is None:
logger.warning_once(
"The attention layers in this model are transitioning from computing the RoPE embeddings internally "
"through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed "
"`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be "
"removed and `position_embeddings` will be mandatory."
)
cos, sin = self.rotary_emb(value_states, position_ids)
else:
cos, sin = position_embeddings

src/transformers/models/qwen2_vl/modeling_qwen2_vl.py Outdated Show resolved Hide resolved
@li-plus li-plus force-pushed the fast-qwen2vl-vision-rope branch from 2e912f3 to 343e47e Compare January 24, 2025 02:01
@li-plus
Copy link
Contributor Author

li-plus commented Jan 24, 2025

I just rebased and the CI is failing on this check. Is it related to this PR?

...
No differences found for 
src/transformers/models/moonshine/modeling_moonshine.py.
Traceback (most recent call last):
  File "/root/transformers/utils/check_modular_conversion.py", line 80, in <module>
    raise ValueError("Some diff and their modeling code did not match.")
ValueError: Some diff and their modeling code did not match.

Exited with code exit status 1

@zucchini-nlp
Copy link
Member

@li-plus Right, you need to add the changes in a "modular_model.py" file and then running make fix-copies will copy everything to the actual "modeling_model.py". So you make CI green by adding the same changes to the modular_qwen2vl.py and then with make fix-copies

zucchini-nlp
zucchini-nlp approved these changes Jan 24, 2025
View reviewed changes
Copy link
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating. Left a small comment and we need to make CI green. Approving and after changes feel free to @ ArthurZucker , the core maintainer

For the core maintainer: not sure if we need to keep the signature in functions (apply_rotary_pos_emb_vision) for BC

src/transformers/models/qwen2_vl/modeling_qwen2_vl.py Outdated Show resolved Hide resolved
src/transformers/models/qwen2_vl/modeling_qwen2_vl.py Show resolved Hide resolved
@li-plus li-plus force-pushed the fast-qwen2vl-vision-rope branch from 343e47e to 64c2827 Compare January 25, 2025 13:57
@li-plus
Copy link
Contributor Author

li-plus commented Jan 25, 2025

I see. It seems that this change also affects the implementation of Qwen2.5VL since it reuses the modules in Qwen2VL. I need to get a Qwen2.5VL model to ensure it's bug-free.

@zucchini-nlp
Copy link
Member

@li-plus running make- fix-copies should fix it and copy content from Qwen2VL to 2.5VL

@li-plus li-plus force-pushed the fast-qwen2vl-vision-rope branch from 64c2827 to 8c22089 Compare January 28, 2025 03:31
@li-plus
Copy link
Contributor Author

li-plus commented Jan 28, 2025

Running make fix-copies ports changes to Qwen2.5VL incompatible with other modules in it. Fortunately just now Qwen2.5VL has been released so I have tested and fixed it. Now this PR is compatible for both Qwen2VL & Qwen2.5VL.

@li-plus
Copy link
Contributor Author

li-plus commented Jan 28, 2025

@ArthurZucker Hi, would you take a look at this PR? It's been approved and the CI turns green.

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zucchini-nlp zucchini-nlp zucchini-nlp approved these changes

@qubvel qubvel qubvel left review comments

Assignees
No one assigned
Labels
Multimodal optimization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@li-plus @qubvel @zucchini-nlp @HuggingFaceDocBuilderDev